3D/4D facial expression analysis: An advanced annotated face model approach

a r t i c l e i n f o Facial expression analysis has interested many researchers in the past decade due to its potential applications in various fields such as human–computer interaction, psychological studies, and facial animation. Three-dimensional facial data has been proven to be insensitive to illumination condition and head pose, and has hence gathered attention in recent years. In this paper, we focus on discrete expression classification using 3D data from the human face. The paper is divided in two parts. In the first part, we present improvement to the fitting of the Annotated Face Model (AFM) so that a dense point correspondence can be found in terms of both position and semantics among static 3D face scans or frames in 3D face sequences. Then, an expression recognition framework on static 3D images is presented. It is based on a Point Distribution Model (PDM) which can be built on different features. In the second part of this article, a systematic pipeline that operates on dynamic 3D sequences (4D datasets or 3D videos) is proposed and alternative modules are investigated as a comparative study. We evaluated both 3D and 4D Facial Expression Recognition pipelines on two publicly available facial expression databases and obtained promising results. Facial expression analysis/recognition 1 (FER) has interested many researchers due to its various purposes and applications. It plays a key role in emotion recognition and can thus, contribute to the development of human–computer interaction systems. It can also be used to improve the performance of face recognition systems by providing prior knowledge on the facial motions and facial feature deformations. This is particularly intriguing considering that the mouth area contains significant amount of discriminative information [1], and yet, is the region where most of the facial deformations occur. Other applications of FER include, but are not limited to, psychological studies, tiredness detection, facial animation, robotics, and virtual reality. Facial expressions are generated by facial muscle contractions which result in temporary facial deformations in both facial geometry and texture. The previous studies have focused primarily on the 2D domain due to the prevalence of data in the relevant modalities (i.e., images and videos). Comprehensive surveys in this area include those by Fasel and Luettin [2], Pantic et al. [3], and Zeng et al. [4]. While these 2D facial expression recognition systems have achieved remarkable performance, challenges in 2D face recognition still present themselves …